Vehicular control system with vehicle trajectory tracking

ABSTRACT

A vehicular control system includes a forward viewing camera and a non-vision sensor. The control, responsive to processing of captured non-vision sensor data and processing of frames of captured image data, at least in part controls the equipped vehicle to travel along a road. The control, responsive at least in part to processing of frames of captured image data, determines lane markers on the road and determines presence of another vehicle traveling along the traffic lane ahead of the equipped vehicle and determines a trajectory of the other vehicle relative to the equipped vehicle. The control stores a trajectory history of the determined trajectory of the other vehicle relative to the equipped vehicle as the other vehicle and the equipped vehicle travel along the traffic lane.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/369,921, filed Dec. 6, 2016, now U.S. Pat. No. 10,407,047,which claims the filing benefits of U.S. provisional application Ser.No. 62/263,888, filed Dec. 7, 2015, which is hereby incorporated hereinby reference in its entirety.

FIELD OF THE INVENTION

The present invention relates generally to a vehicle driver assistancesystem for a vehicle and, more particularly, to a vehicle driverassistance system that utilizes one or more cameras at a vehicle.

BACKGROUND OF THE INVENTION

Use of imaging sensors in vehicle imaging systems is common and known.Examples of such known systems are described in U.S. Pat. Nos.5,949,331; 5,670,935 and/or 5,550,677, which are hereby incorporatedherein by reference in their entireties.

SUMMARY OF THE INVENTION

The present invention provides a driver assistance system or visionsystem or imaging system for a vehicle that utilizes one or more cameras(preferably one or more CMOS cameras) to capture image datarepresentative of images exterior of the vehicle, and a control,responsive to processing of captured image data (and/or other datacaptured by a sensor or sensors of the vehicle), determines the presenceand motion of a target vehicle in the field of view/sensing of thesensor, and determines the path of travel of the other target vehiclerelative to the current location of the equipped vehicle as the equippedvehicle moves along its own path of travel. The control translates thedetermined location and motion of the target vehicle to a localcoordinate system of the equipped vehicle so that the path of travel ofthe target vehicle is relative to the path of travel of the equippedvehicle.

Self-driving vehicles will require multiple methods of determining theappropriate trajectory to follow. Depending on the driving situation,there may be limited information available on how to navigate. Themotion of other vehicles can indicate an available trajectory for thesubject vehicle to take. The techniques of the present invention enablerecording the trajectory that one or more target vehicles have traveled.This trajectory is updated continuously based on the motion of thesubject vehicle to remain accurate to the subject vehicle's relativeframe of reference. These techniques can utilize vehicle dynamicssensors data or information such as yaw rate and vehicle speed. Includedin these techniques is correcting for noise in the sensors to allow forbetter vehicle tracking. Linking data from multiple disparate sensingtechnologies can reduce the errors associated with determining thetrajectory of the subject and target vehicles. This data can be used toenhance the operation of many automotive features, such as, for example,traffic jam assist, highway pilot, automated taxi service, and/or thelike.

These and other objects, advantages, purposes and features of thepresent invention will become apparent upon review of the followingspecification in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a plan view of a vehicle with a control system thatincorporates cameras and/or other sensors in accordance with the presentinvention;

FIG. 2 is a schematic of the system of the present invention;

FIG. 3 is a schematic showing the angle and coordinate system of atarget vehicle relative to the subject vehicle in accordance with thepresent invention;

FIGS. 4-6 are schematics showing motion of the target vehicle andtransformation of the motion points in accordance with the presentinvention;

FIG. 7 is a graph of a circle for determining curvature using linearapproximations and calculus methods in accordance with the presentinvention; and

FIG. 8 is a graph of another circle for determining curvature using twopoints on the circle and a tangent in accordance with the presentinvention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

A vehicle vision system and/or driver assist system and/or objectdetection system and/or alert system operates to capture images exteriorof the vehicle and may process the captured image data to display imagesand to detect objects at or near the vehicle and in the predicted pathof the vehicle, such as to assist a driver of the vehicle in maneuveringthe vehicle in a rearward direction. The vision system includes an imageprocessor or image processing system that is operable to receive imagedata from one or more cameras and provide an output to a display devicefor displaying images representative of the captured image data.Optionally, the vision system may provide display, such as a rearviewdisplay or a top down or bird's eye or surround view display or thelike.

Referring now to the drawings and the illustrative embodiments depictedtherein, a vehicle 10 (FIG. 1) includes a control system 12 thatincludes at least one exterior facing imaging sensor or camera, such asa forward facing camera 14 a at the windshield of the vehicle and/or aforward facing camera 14 b at a forward portion of the vehicle, and arearward facing imaging sensor or camera 14 c at the rear of thevehicle, and one or more sidewardly/rearwardly facing cameras 14 d atrespective sides of the vehicle, which capture images or image dataexterior of the vehicle, with the cameras each having a lens forfocusing images at or onto an imaging array or imaging plane or imagerof the camera. The forward viewing camera may be disposed at thewindshield of the vehicle and view through the windshield and forward ofthe vehicle, such as for a machine vision system (such as for trafficsign recognition, headlamp control, pedestrian detection, collisionavoidance, lane marker detection and/or the like). The system 12includes other sensors, such as a radar sensor or sensors 16, a lidarsensor or sensors 18 and/or a yaw rate sensor 20. A GPS system andantenna 22 is also included in the control system. The control systemincludes a sensor processor 24 and central processor 26. The sensorprocessor may process image data captured by the camera or cameras andsensor data captured or generated by the other sensors, such as todetect objects or the like. The data transfer or signal communicationfrom the camera to the ECU may comprise any suitable data orcommunication link, such as a vehicle network bus or the like of theequipped vehicle.

The basis of the system of the present invention is to store a targetvehicle's positions in history. These positions are recorded relative tothe subject vehicle responsive to data captured by one or more sensorsof the subject or equipped vehicle. A coordinate transform may beapplied to transfer the recorded positions to a global reference. Forthe purpose of the methods described below, these positions are insteadconstantly updated so that they remain in the subject vehicle's localcoordinate system even as the subject vehicle moves. This system mayoperate independently without other object or feature tracking methods.For example, this system does not rely on lane markings, road textures,GPS location or high definition maps. Also, additional SLAM

(Simultaneous Localization and Mapping) techniques are not necessary.However, this system can be used in conjunction with those techniquesfor additional information and redundancy. The resulting trajectory thatcan be generated from the recorded points can be used by a self-drivingfeature's lateral motion controller or other features.

The input from the subject vehicle's sensors may require noise filteringand correction (such as by utilizing aspects of the systems described inU.S. Pat. No. 8,694,224, which is hereby incorporated herein byreference in its entirety).

A Kalman filter is used to predict states of the system and measurementsare used to update these predictions. A discrete time Kalman filter isdescribed for use with the methods here to estimate the system states.The target vehicle is modelled as a system with a 4 dimensional vector[dx, dy, Vx, Vy] where dx, dy are the longitudinal and lateral relativepositions of the target vehicle (relative to the subject vehicle) andVx, Vy are the longitudinal and lateral relative velocities of thetarget vehicle (relative to the subject vehicle).

A tuning process is used to optimize the Kalman filter for fastconvergence utilizing known techniques. For example, the measurementnoise covariance R is a calibratable value which can be pre-calculatedoffline. Another embodiment can have the R value determined in realtime. A large number of readings of a known ground truth state aretaken, from which the variance(s) can be calculated. This matrix is keptconstant and matrix Q is varied.

For the purpose of the methods described herein, the history positionsare stored in a FIFO (First In First Out) queue. A number of positionsare stored so a trajectory can be generated even at high speeds or whenthe target vehicle is far away, for calibratable number of positions,such as, for example, 100 positions or thereabouts. To make sure thesepositions are meaningful, and not clustered together, a position isrecorded only when it is at least a calibratable minimum distance fromthe previously recorded position, such as, for example, about 0.5 metersfrom the previously recorded position. The oldest positions are removedfrom the buffer when the maximum buffer size is reached.

When this data is collected or discarded can be determined by variousfactors, specific to the needs of the features being supported by thissystem. For example, the data buffer can be reset when the targetvehicle is no longer detected by the sensors. Also, data may becollected as soon as a target vehicle is detected and stored even if itis not currently being immediately consumed or used by another feature.For example, the system may be used in an autonomous vehicle to track apreceding vehicle's trajectory constantly. In normal operation, theautonomous vehicle would be using other information such as lanemarkings or GPS points to control its steering, but if that informationwas suddenly lost (such as when dirt or snow may cover lane markings onthe road or in close traffic jam driving situations where the vehicleahead of the subject vehicle interferes with the system's ability toview the lane markers), then the system may switch over to following thepreceding vehicle's trajectory as a form of degraded operation.Optionally, in a traffic jam driving situation, the informationpertaining to the preceding vehicle's trajectory may be the primary formof steering control of a steering system of the equipped vehicle (andoptionally for controlling a braking system of the equipped vehicle andan accelerator system of the equipped vehicle) to either follow thedetermined relative path of travel of the target vehicle or to avoid thetarget vehicle when the system determines that the relative path oftravel of the target vehicle may interfere with or intercept the currentpath of travel of the equipped vehicle.

The buffer stores the following information about the leading or targetvehicle (see FIG. 3).

dx=relative longitudinal position from target vehicle,

dy=relative lateral position from target vehicle, and

dθ=mathematical slope between two consecutive points in buffer.

Motion Tracking is responsible for generating a trajectory that thetarget vehicle will follow. Before the new point is recorded at a timeinstant, the target vehicle history buffer data is updated based on thesubject vehicle's motion. With reference to FIGS. 4-6, at the timeinstance t, if the subject vehicle is at position A, then at next timeinstant t+1, the subject vehicle will move to a position A′ (FIG. 4).The position update can be performed when another position is added fromthe latest sensor input or the system references the vehicle's positionhistory. The sensor provides the following inputs by tracking the targetvehicle: relative longitudinal position dx and relative lateral positiondy.

To resolve the relative position data with the motion of the subjectvehicle, the system translates the old position data (dx_(old),dy_(old)) to the new coordinates (dx_(new), dy_(new)) in the newreference system. For the purpose of the methods described herein, thismotion will be determined from the subject vehicle's velocity and yawrate. The origin of subject vehicle shifts to (a, b) in single time stepwith a rotation of angle θ. This motion and resulting transformation isshown in FIGS. 4-6.

This is calculated as follows:a=Vx*Δtb=Vy*Δtø={dot over (ψ)}*Δtwhere:

Vx=Longitudinal velocity of subject vehicle Vy=Lateral velocity ofsubject vehicle

{dot over (ψ)}=Yaw Rate of subject vehicle Δt=Sample time

Another embodiment can use subject vehicle acceleration from the subjectvehicle's inertial measurement unit (IMU).

All previous target vehicle position coordinate values in the buffer aretransformed using following relations:

dx_(new) = (dx_(old) − a) * cos  ⌀ + (dy_(old) − b) * sin  ⌀dy_(new) = −1 * (dx_(old) − a) * sin  ⌀ + (dy_(old) − b) * cos  ⌀${d\;\theta} = {\tan^{- 1}\frac{\Delta\;{dy}_{new}}{\Delta\;{dx}_{new}}}$where Δdy_(new) and Δdx_(new) is the difference of the lateral andlongitudinal positions of two consecutive points in the buffer.

There are many possible outputs from the motion tracking module. Forexample, the module or system can output information about a point onthe trajectory ahead for a lateral control module to use to steer thesubject vehicle. Optionally, for example, the outputs may be dy (lateralerror from required trajectory), r (heading angle) and K (trajectorycurvature). The target reference point on the trajectory is selectedfrom the buffer based on the velocity of the subject vehicle tocalculate a relative position ahead from a predetermined look aheadtime, such as, for example, a look ahead time of about 0.5 seconds. If apoint at the look ahead time does not exist then the farthest pointahead will be used instead. The dy and dθ are chosen corresponding tothis point. The angle dθ corresponds to the heading angle r at thetarget reference point.

Different methods can be used to determine curvature, such as by usinglinear approximations and calculus methods, or finding a radius of acurve using three points along the curve or using two points along thecurve and a tangent line at the curve, as discussed below.

Using Linear Approximations and Calculus Methods:

Let (x₁, y₁), (x₂, y₂) and (x₃, y₃) be the coordinates of point P1, P2and P3 respectively.

$\frac{dy}{dx}\mspace{14mu}{is}\mspace{14mu}{calculated}\mspace{14mu}{using}\text{:}$m 1 = Slope  joining  point  1  and  2m 2 = Slope  joining  points  2  and  3$\frac{dy}{dx} = {{average}\mspace{14mu}{of}\mspace{14mu}{these}\mspace{14mu}{slopes}}$

The second derivative of slope is approximated as

$\frac{\Delta\; m}{\Delta\; x} = {\frac{{m\; 2} - {m\; 1}}{\frac{\left( {{x\; 2} + {x\; 1}} \right)}{2} - \frac{{x\; 3} + {x\; 2}}{2}}\text{\textasciitilde}\;\frac{d^{2}y}{d\; x^{2}}}$${{Trajectory}\mspace{14mu}{Radius}\mspace{14mu} R} = \frac{\left\lbrack {1 + \left( \frac{dy}{dx} \right)^{2}} \right\rbrack^{3/2}}{\frac{d^{2}y}{{dx}^{2}}}$$K = \frac{1}{R}$Finding the Radius of the Circle Using 3 Points:

Let (x₁, y₁), (x₂, y₂) and (x₃, y₃) be the coordinates of point P1, P2and P3 respectively (see FIG. 7).

The center of the circle at point (xc, yc) is:

${xc} = \frac{{m\; 1\; m\; 2\left( {y_{1} - y_{3}} \right)} + {m\; 2\left( {x_{1} + x_{2}} \right)} - {m\; 1\left( {x_{2} + x_{3}} \right)}}{2\;\left( {{m\; 2} - {m\; 1}} \right)}$${yc} = {{{- \frac{1}{m\; 1}}\left( {{xc} - \frac{x_{1} + x_{2}}{2}} \right)} + \frac{y_{1} + y_{2}}{2}}$

And the trajectory radius is found using this center and any one pointusing distance formula:

${{Trajectory}\mspace{14mu}{Radius}\mspace{14mu} R} = \sqrt[2]{\left( {{xc} - x_{1}} \right)^{2} + \left( {{yc} - y_{1}} \right)^{2}}$$K = \frac{1}{R}$Curvature from Two Points and a Tangent:

Consider two points as P1 and P2 (see FIG. 8). The line from center ofcircle to P1 at the tangent is perpendicular to the tangent. The centerwill be equidistant from P1 and P2.

The equation of the tangent line is ax+by=c, where a=−m1 (Slope at pointP1), b=1, c=ax₁+by₁. (x₁, y₁) and (x₂, y₂) are the coordinates of pointP1 and P2 respectively.

The equation of the perpendicular line passing through P1 is:bx−ay=bx ₁ −ay ₁

The equation of a line that is perpendicular to the line joining P1 andP2 and that passes through the midpoint of that line is:

${{\left( {x_{1} - x_{2}} \right)x} + {\left( {y_{1} - y_{2}} \right)y}} = {\frac{1}{2}*\left( {x_{1}^{2} + y_{1}^{2} - x_{2}^{2} - y_{2}^{2}} \right)}$

The solution of this equation can be gained by standard methods. Forexample, using Cramer's rule:

${{Coefficient}\mspace{14mu}{Matrix}\mspace{14mu} D} = \begin{bmatrix}b & {- a} \\\left( {x_{1} - x_{2}} \right) & \left( {y_{1} - y_{2}} \right)\end{bmatrix}$ $\;{{X - {{matrix}\mspace{11mu}{Dx}}} = \begin{bmatrix}{{bx}_{1} - {ay}_{1}} & {- a} \\{\frac{1}{2}*\left( {x_{1}^{2} + y_{1}^{2} - x_{2}^{2} - y_{2}^{2}} \right)} & \left( {y_{1} - y_{2}} \right)\end{bmatrix}}$ ${Y - {{matrix}\mspace{14mu}{Dy}}} = \begin{bmatrix}b & {{bx}_{1} - {ay}_{1}} \\\left( {x_{1} - x_{2}} \right) & {\frac{1}{2}*\left( {x_{1}^{2} + y_{1}^{2} - x_{2}^{2} - y_{2}^{2}} \right)}\end{bmatrix}$

The center is at point (xc, yc):

${xc} = \frac{{Dx}}{D}$ ${yc} = \frac{{Dy}}{D}$

And the trajectory radius is found using this center and any one pointusing distance formula:

${{Trajectory}\mspace{14mu}{Radius}\mspace{14mu} R} = \sqrt[2]{\left( {{xc} - x_{1}} \right)^{2} + \left( {{yc} - y_{1}} \right)^{2}}$$K = \frac{1}{R}$

Finally, for the purpose of smoothing the output of the motion trackingmodule, other points in the buffer may be used beyond those in themethods described above. For example, the relative heading can be anaverage of the recorded headings between the selected point ahead andall the closer points.

Therefore, the present invention includes sensors (such as image sensorsor cameras and/or radar sensors and/or lidar sensors) to capture dataand a processor processes captured data to determine the presence andmovement of another vehicle (such as another vehicle ahead of thesubject vehicle and traveling along the same road as the subjectvehicle). The system tracks the other vehicle and determines themovement or path of the other vehicle relative to the subject vehicle.This is done by translating the actual location coordinates of thedetermined other vehicle to a local coordinate system of the subjectvehicle (where the subject vehicle's current location is always theorigin of the coordinate system, with the origin moving with the subjectvehicle). The system determines the path of travel of the other vehiclerelative to the subject vehicle, and can use this information whenautonomously or semi-autonomously controlling the subject vehicle tofollow the determined other vehicle.

The system may utilize sensors, such as radar or lidar sensors or thelike. The sensing system may utilize aspects of the systems described inU.S. Pat. Nos. 8,027,029; 8,013,780; 6,825,455; 7,053,357; 7,408,627;7,405,812; 7,379,163; 7,379,100; 7,375,803; 7,352,454; 7,340,077;7,321,111; 7,310,431; 7,283,213; 7,212,663; 7,203,356; 7,176,438;7,157,685; 6,919,549; 6,906,793; 6,876,775; 6,710,770; 6,690,354;6,678,039; 6,674,895 and/or 6,587,186, and/or International PublicationNo. WO 2011/090484 and/or U.S. Publication No. US-2010-0245066 and/orU.S. provisional applications, Ser. No. 62/375,161, filed Aug. 15, 2016,Ser. No. 62/361,586, filed Jul. 13, 2016, Ser. No. 62/359,913, filedJul. 8, 2016, Ser. No. 62/349,874, filed Jun. 14, 2016, Ser. No.62/330,557, filed May 2, 2016, Ser. No. 62/313,279, filed Mar. 25, 2016,Ser. No. 62/303,546, filed Mar. 4, 2016, and/or Ser. No. 62/289,441,filed Feb. 1, 2016, which are hereby incorporated herein by reference intheir entireties.

The camera or sensor may comprise any suitable camera or sensor.Optionally, the camera may comprise a “smart camera” that includes theimaging sensor array and associated circuitry and image processingcircuitry and electrical connectors and the like as part of a cameramodule, such as by utilizing aspects of the vision systems described inInternational Publication Nos. WO 2013/081984 and/or WO 2013/081985,which are hereby incorporated herein by reference in their entireties.

The system includes an image processor operable to process image datacaptured by the camera or cameras, such as for detecting objects orother vehicles or pedestrians or the like in the field of view of one ormore of the cameras. For example, the image processor may comprise animage processing chip selected from the EYEQ family of image processingchips available from Mobileye Vision Technologies Ltd. of Jerusalem,Israel, and may include object detection software (such as the typesdescribed in U.S. Pat. Nos. 7,855,755; 7,720,580 and/or 7,038,577, whichare hereby incorporated herein by reference in their entireties), andmay analyze image data to detect vehicles and/or other objects.Responsive to such image processing, and when an object or other vehicleis detected, the system may generate an alert to the driver of thevehicle and/or may generate an overlay at the displayed image tohighlight or enhance display of the detected object or vehicle, in orderto enhance the driver's awareness of the detected object or vehicle orhazardous condition during a driving maneuver of the equipped vehicle.

The vehicle may include any type of sensor or sensors, such as imagingsensors or radar sensors or lidar sensors or ladar sensors or ultrasonicsensors or the like. The imaging sensor or camera may capture image datafor image processing and may comprise any suitable camera or sensingdevice, such as, for example, a two dimensional array of a plurality ofphotosensor elements arranged in at least 640 columns and 480 rows (atleast a 640×480 imaging array, such as a megapixel imaging array or thelike), with a respective lens focusing images onto respective portionsof the array. The photosensor array may comprise a plurality ofphotosensor elements arranged in a photosensor array having rows andcolumns. Preferably, the imaging array has at least 300,000 photosensorelements or pixels, more preferably at least 500,000 photosensorelements or pixels and more preferably at least 1 million photosensorelements or pixels. The imaging array may capture color image data, suchas via spectral filtering at the array, such as via an RGB (red, greenand blue) filter or via a red/red complement filter or such as via anRCC (red, clear, clear) filter or the like. The logic and controlcircuit of the imaging sensor may function in any known manner, and theimage processing and algorithmic processing may comprise any suitablemeans for processing the images and/or image data.

For example, the vision system and/or processing and/or camera and/orcircuitry may utilize aspects described in U.S. Pat. Nos. 9,233,641;9,146,898; 9,174,574; 9,090,234; 9,077,098; 8,818,042; 8,886,401;9,077,962; 9,068,390; 9,140,789; 9,092,986; 9,205,776; 8,917,169;8,694,224; 7,005,974; 5,760,962; 5,877,897; 5,796,094; 5,949,331;6,222,447; 6,302,545; 6,396,397; 6,498,620; 6,523,964; 6,611,202;6,201,642; 6,690,268; 6,717,610; 6,757,109; 6,802,617; 6,806,452;6,822,563; 6,891,563; 6,946,978; 7,859,565; 5,550,677; 5,670,935;6,636,258; 7,145,519; 7,161,616; 7,230,640; 7,248,283; 7,295,229;7,301,466; 7,592,928; 7,881,496; 7,720,580; 7,038,577; 6,882,287;5,929,786 and/or 5,786,772, and/or U.S. Publication Nos.US-2014-0340510; US-2014-0313339; US-2014-0347486; US-2014-0320658;US-2014-0336876; US-2014-0307095; US-2014-0327774; US-2014-0327772;US-2014-0320636; US-2014-0293057; US-2014-0309884; US-2014-0226012;US-2014-0293042; US-2014-0218535; US-2014-0218535; US-2014-0247354;US-2014-0247355; US-2014-0247352; US-2014-0232869; US-2014-0211009;US-2014-0160276; US-2014-0168437; US-2014-0168415; US-2014-0160291;US-2014-0152825; US-2014-0139676; US-2014-0138140; US-2014-0104426;US-2014-0098229; US-2014-0085472; US-2014-0067206; US-2014-0049646;US-2014-0052340; US-2014-0025240; US-2014-0028852; US-2014-005907;US-2013-0314503; US-2013-0298866; US-2013-0222593; US-2013-0300869;US-2013-0278769; US-2013-0258077; US-2013-0258077; US-2013-0242099;US-2013-0215271; US-2013-0141578 and/or US-2013-0002873, which are allhereby incorporated herein by reference in their entireties. The systemmay communicate with other communication systems via any suitable means,such as by utilizing aspects of the systems described in InternationalPublication Nos. WO/2010/144900; WO 2013/043661 and/or WO 2013/081985,and/or U.S. Pat. No. 9,126,525, which are hereby incorporated herein byreference in their entireties.

The camera module and circuit chip or board and imaging sensor may beimplemented and operated in connection with various vehicularvision-based systems, and/or may be operable utilizing the principles ofsuch other vehicular systems, such as a vehicle headlamp control system,such as the type disclosed in U.S. Pat. Nos. 5,796,094; 6,097,023;6,320,176; 6,559,435; 6,831,261; 7,004,606; 7,339,149 and/or 7,526,103,which are all hereby incorporated herein by reference in theirentireties, a rain sensor, such as the types disclosed in commonlyassigned U.S. Pat. Nos. 6,353,392; 6,313,454; 6,320,176 and/or7,480,149, which are hereby incorporated herein by reference in theirentireties, a vehicle vision system, such as a forwardly, sidewardly orrearwardly directed vehicle vision system utilizing principles disclosedin U.S. Pat. Nos. 5,550,677; 5,670,935; 5,760,962; 5,877,897; 5,949,331;6,222,447; 6,302,545; 6,396,397; 6,498,620; 6,523,964; 6,611,202;6,201,642; 6,690,268; 6,717,610; 6,757,109; 6,802,617; 6,806,452;6,822,563; 6,891,563; 6,946,978 and/or 7,859,565, which are all herebyincorporated herein by reference in their entireties, a trailer hitchingaid or tow check system, such as the type disclosed in U.S. Pat. No.7,005,974, which is hereby incorporated herein by reference in itsentirety, a reverse or sideward imaging system, such as for a lanechange assistance system or lane departure warning system or for a blindspot or object detection system, such as imaging or detection systems ofthe types disclosed in U.S. Pat. Nos. 7,881,496; 7,720,580; 7,038,577;5,929,786 and/or 5,786,772, which are hereby incorporated herein byreference in their entireties, a video device for internal cabinsurveillance and/or video telephone function, such as disclosed in U.S.Pat. Nos. 5,760,962; 5,877,897; 6,690,268 and/or 7,370,983, and/or U.S.Publication No. US-2006-0050018, which are hereby incorporated herein byreference in their entireties, a traffic sign recognition system, asystem for determining a distance to a leading or trailing vehicle orobject, such as a system utilizing the principles disclosed in U.S. Pat.Nos. 6,396,397 and/or 7,123,168, which are hereby incorporated herein byreference in their entireties, and/or the like.

Optionally, the vision system may include a display for displayingimages captured by one or more of the imaging sensors for viewing by thedriver of the vehicle while the driver is normally operating thevehicle. Optionally, for example, the vision system may include a videodisplay device, such as by utilizing aspects of the video displaysystems described in U.S. Pat. Nos. 5,530,240; 6,329,925; 7,855,755;7,626,749; 7,581,859; 7,446,650; 7,338,177; 7,274,501; 7,255,451;7,195,381; 7,184,190; 5,668,663; 5,724,187; 6,690,268; 7,370,983;7,329,013; 7,308,341; 7,289,037; 7,249,860; 7,004,593; 4,546,551;5,699,044; 4,953,305; 5,576,687; 5,632,092; 5,677,851; 5,708,410;5,737,226; 5,802,727; 5,878,370; 6,087,953; 6,173,508; 6,222,460;6,513,252 and/or 6,642,851, and/or U.S. Publication Nos.US-2012-0162427; US-2006-0050018 and/or US-2006-0061008, which are allhereby incorporated herein by reference in their entireties.

Optionally, the vision system (utilizing the forward facing camera and arearward facing camera and other cameras disposed at the vehicle withexterior fields of view) may be part of or may provide a display of atop-down view or birds-eye view system of the vehicle or a surround viewat the vehicle, such as by utilizing aspects of the vision systemsdescribed in International Publication Nos. WO 2010/099416; WO2011/028686; WO 2012/075250; WO 2013/019795; WO 2012/075250; WO2012/145822; WO 2013/081985; WO 2013/086249 and/or WO 2013/109869,and/or U.S. Publication No. US-2012-0162427, which are herebyincorporated herein by reference in their entireties.

Changes and modifications in the specifically described embodiments canbe carried out without departing from the principles of the invention,which is intended to be limited only by the scope of the appendedclaims, as interpreted according to the principles of patent lawincluding the doctrine of equivalents.

The invention claimed is:
 1. A vehicular control system, said vehicularcontrol system comprising: a forward viewing camera disposed at awindshield of a vehicle equipped with said vehicular control system,wherein said camera has a field of view through the windshield andforward of the equipped vehicle, wherein said camera is operable tocapture frames of image data; at least one non-vision sensor disposed atthe equipped vehicle so as to have a field of sensing exterior of theequipped vehicle, wherein said at least one non-vision sensor isoperable to capture non-vision sensor data; a control comprising atleast one processor; wherein frames of image data captured by saidcamera are processed at said control; wherein non-vision sensor datacaptured by said at least one non-vision sensor is processed at saidcontrol; wherein said control, responsive to processing capturednon-vision sensor data at said control and processing frames of capturedimage data at said control, at least in part controls the equippedvehicle as the equipped vehicle travels along a road; wherein saidcontrol, responsive at least in part to processing frames of capturedimage data at said control, determines presence of lane markers of atraffic lane along which the equipped vehicle is traveling on the road;wherein, as the equipped vehicle travels along the road, and at least inpart via processing frames of captured image data at said control, saidcontrol determines presence of another vehicle traveling along thetraffic lane ahead of the equipped vehicle and determines a trajectoryof the other vehicle relative to the equipped vehicle; wherein saidcontrol transforms the determined trajectory of the other vehicle to alocal coordinate system of the equipped vehicle, and wherein saidcontrol updates the local coordinate system of the equipped vehicle asthe equipped vehicle travels along the road; wherein said control storesa trajectory history of the transformed determined trajectory of theother vehicle relative to the equipped vehicle as the other vehicle andthe equipped vehicle travel along the traffic lane; and wherein,responsive to determination of presence of lane markers on the road viaprocessing of frames of captured image data being compromised, saidcontrol uses the stored trajectory history to at least in part controlthe equipped vehicle.
 2. The vehicular control system of claim 1,wherein, responsive to determination of presence of lane markers on theroad via processing of frames of captured image data being compromised,said control uses the stored trajectory history to at least in partcontrol the equipped vehicle to follow the other vehicle.
 3. Thevehicular control system of claim 1, wherein said control determines thetrajectory of the other vehicle relative to the equipped vehicleresponsive at least in part to determination of changes in location ofthe other vehicle relative to the equipped vehicle.
 4. The vehicularcontrol system of claim 1, wherein said control determines thetrajectory of the other vehicle relative to the equipped vehicleresponsive at least in part to determination of changes in speed of theother vehicle relative to the equipped vehicle.
 5. The vehicular controlsystem of claim 1, wherein said control controls the equipped vehicle bycontrolling steering of the equipped vehicle, braking of the equippedvehicle and acceleration of the equipped vehicle.
 6. The vehicularcontrol system of claim 1, wherein said control controls the equippedvehicle by controlling at least one selected from the group consistingof (i) steering of the equipped vehicle, (ii) braking of the equippedvehicle and (iii) acceleration of the equipped vehicle.
 7. The vehicularcontrol system of claim 6, wherein said control controls the equippedvehicle to follow the other vehicle.
 8. The vehicular control system ofclaim 1, wherein, responsive to processing frames of captured image dataat said control, said control determines a plurality of positions of theother vehicle relative to the equipped vehicle.
 9. The vehicular controlsystem of claim 8, wherein the plurality of positions are stored inmemory.
 10. The vehicular control system of claim 9, wherein saidcontrol stores a position of the other vehicle relative to the equippedvehicle only when the determined position of the other vehicle relativeto the equipped vehicle is at least a threshold distance from apreviously stored position of the other vehicle relative to the equippedvehicle.
 11. The vehicular control system of claim 1, wherein saidcontrol is operable to control the equipped vehicle at least in partresponsive to a mapping system.
 12. The vehicular control system ofclaim 11, wherein, responsive to loss of mapping data associated withthe mapping system, said control controls the equipped vehicle to followthe other vehicle.
 13. The vehicular control system of claim 1, whereinsaid control, responsive at least in part to processing frames ofcaptured image data at said control, is operable to determine apedestrian present within the field of view of said camera, and whereinsaid control at least in part controls the equipped vehicle at least inpart responsive to determination of a pedestrian within the field ofview of said camera.
 14. The vehicular control system of claim 1,wherein said at least one non-vision sensor comprises at least one radarsensor.
 15. The vehicular control system of claim 1, wherein said atleast one non-vision sensor comprises at least one lidar sensor.
 16. Avehicular control system, said vehicular control system comprising: aforward viewing camera disposed at a windshield of a vehicle equippedwith said vehicular control system, wherein said camera has a field ofview through the windshield and forward of the equipped vehicle, whereinsaid camera is operable to capture frames of image data; at least oneradar sensor disposed at the equipped vehicle so as to have a field ofsensing exterior of the equipped vehicle, wherein said at least oneradar sensor is operable to capture radar sensor data; a controlcomprising at least one processor; wherein frames of image data capturedby said camera are processed at said control; wherein radar sensor datacaptured by said at least one radar sensor is processed at said control;wherein said control, responsive to processing captured radar sensordata at said control and processing frames of captured image data atsaid control, at least in part controls the equipped vehicle as theequipped vehicle travels along a road; wherein said control controls theequipped vehicle by controlling at least one selected from the groupconsisting of (i) steering of the equipped vehicle, (ii) braking of theequipped vehicle and (iii) acceleration of the equipped vehicle; whereinsaid control, responsive at least in part to processing frames ofcaptured image data at said control, determines presence of lane markersof a traffic lane along which the equipped vehicle is traveling on theroad; wherein, as the equipped vehicle travels along the road, and atleast in part via processing frames of captured image data at saidcontrol, said control determines presence of another vehicle travelingalong the traffic lane ahead of the equipped vehicle and determines atrajectory of the other vehicle relative to the equipped vehicle;wherein said control transforms the determined trajectory of the othervehicle to a local coordinate system of the equipped vehicle, andwherein said control updates the local coordinate system of the equippedvehicle as the equipped vehicle travels along the road; and wherein saidcontrol stores a trajectory history of the transformed determinedtrajectory of the other vehicle relative to the equipped vehicle as theother vehicle and the equipped vehicle travel along the traffic lane.17. The vehicular control system of claim 16, wherein, responsive todetermination of presence of lane markers on the road via processing offrames of captured image data being compromised, said control uses thestored trajectory history to at least in part control the equippedvehicle.
 18. The vehicular control system of claim 17, wherein saidcontrol controls the equipped vehicle by controlling steering of theequipped vehicle, braking of the equipped vehicle and acceleration ofthe equipped vehicle.
 19. The vehicular control system of claim 17,wherein, responsive to determination of presence of lane markers on theroad via processing of frames of captured image data being compromised,said control uses the stored trajectory history to at least in partcontrol the equipped vehicle to follow the other vehicle.
 20. Thevehicular control system of claim 17, wherein, responsive to processingframes of captured image data at said control, said control determines aplurality of positions of the other vehicle relative to the equippedvehicle, and wherein the plurality of positions are stored in memory,and wherein said control stores a position of the other vehicle relativeto the equipped vehicle only when the determined position of the othervehicle relative to the equipped vehicle is at least a thresholddistance from a previously stored position of the other vehicle relativeto the equipped vehicle.
 21. The vehicular control system of claim 16,wherein said control determines the trajectory of the other vehiclerelative to the equipped vehicle responsive at least in part todetermination of changes in at least one selected from the groupconsisting of (i) location of the other vehicle relative to the equippedvehicle and (ii) speed of the other vehicle relative to the equippedvehicle.
 22. The vehicular control system of claim 16, wherein saidcontrol is operable to control the equipped vehicle at least in partresponsive to a mapping system, and wherein, responsive to loss ofmapping data associated with the mapping system, said control controlsthe equipped vehicle to follow the other vehicle.
 23. The vehicularcontrol system of claim 16, wherein said control, responsive at least inpart to processing frames of captured image data at said control, isoperable to determine a pedestrian present within the field of view ofsaid camera, and wherein said control at least in part controls theequipped vehicle at least in part responsive to determination of apedestrian within the field of view of said camera.
 24. A vehicularcontrol system, said vehicular control system comprising: a forwardviewing camera disposed at a windshield of a vehicle equipped with saidvehicular control system, wherein said camera has a field of viewthrough the windshield and forward of the equipped vehicle, wherein saidcamera is operable to capture frames of image data; at least one lidarsensor disposed at the equipped vehicle so as to have a field of sensingexterior of the equipped vehicle, wherein said at least one lidar sensoris operable to capture lidar sensor data; a control comprising at leastone processor; wherein frames of image data captured by said camera areprocessed at said control; wherein lidar sensor data captured by said atleast one lidar sensor is processed at said control; wherein saidcontrol, responsive to processing captured lidar sensor data at saidcontrol and processing frames of captured image data at said control, atleast in part controls the equipped vehicle as the equipped vehicletravels along a road; wherein said control controls the equipped vehicleby controlling at least one selected from the group consisting of (i)steering of the equipped vehicle, (ii) braking of the equipped vehicleand (iii) acceleration of the equipped vehicle; wherein said control,responsive at least in part to processing frames of captured image dataat said control, determines presence of lane markers of a traffic lanealong which the equipped vehicle is traveling on the road; wherein, asthe equipped vehicle travels along the road, and at least in part viaprocessing frames of captured image data at said control, said controldetermines presence of another vehicle traveling along the traffic laneahead of the equipped vehicle and determines a trajectory of the othervehicle relative to the equipped vehicle; wherein said controltransforms the determined trajectory of the other vehicle to a localcoordinate system of the equipped vehicle, and wherein said controlupdates the local coordinate system of the equipped vehicle as theequipped vehicle travels along the road; and wherein said control storesa trajectory history of the transformed determined trajectory of theother vehicle relative to the equipped vehicle as the other vehicle andthe equipped vehicle travel along the traffic lane.
 25. The vehicularcontrol system of claim 24, wherein, responsive to determination ofpresence of lane markers on the road via processing of frames ofcaptured image data being compromised, said control uses the storedtrajectory history to at least in part control the equipped vehicle. 26.The vehicular control system of claim 25, wherein said control controlsthe equipped vehicle by controlling steering of the equipped vehicle,braking of the equipped vehicle and acceleration of the equippedvehicle.
 27. The vehicular control system of claim 25, wherein,responsive to determination of presence of lane markers on the road viaprocessing of frames of captured image data being compromised, saidcontrol uses the stored trajectory history to at least in part controlthe equipped vehicle to follow the other vehicle.
 28. The vehicularcontrol system of claim 25, wherein, responsive to processing frames ofcaptured image data at said control, said control determines a pluralityof positions of the other vehicle relative to the equipped vehicle, andwherein the plurality of positions are stored in memory, and whereinsaid control stores a position of the other vehicle relative to theequipped vehicle only when the determined position of the other vehiclerelative to the equipped vehicle is at least a threshold distance from apreviously stored position of the other vehicle relative to the equippedvehicle.
 29. The vehicular control system of claim 24, wherein saidcontrol determines the trajectory of the other vehicle relative to theequipped vehicle responsive at least in part to determination of changesin at least one selected from the group consisting of (i) location ofthe other vehicle relative to the equipped vehicle and (ii) speed of theother vehicle relative to the equipped vehicle.
 30. The vehicularcontrol system of claim 24, wherein said control is operable to controlthe equipped vehicle at least in part responsive to a mapping system,and wherein, responsive to loss of mapping data associated with themapping system, said control controls the equipped vehicle to follow theother vehicle.
 31. The vehicular control system of claim 24, whereinsaid control, responsive at least in part to processing frames ofcaptured image data at said control, is operable to determine apedestrian present within the field of view of said camera, and whereinsaid control at least in part controls the equipped vehicle at least inpart responsive to determination of a pedestrian within the field ofview of said camera.